-
Notifications
You must be signed in to change notification settings - Fork 41
Implement Backups & Restore #342
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Conversation
… overall matrix of integration runs
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I have left a few comments on how you managed list backups for an index and list all backups for a project since its not clear to me.
Rest looks good, thanks for making this change!!
*/ | ||
export interface ListBackupsOptions { | ||
/** | ||
* The index name to list backups for. If not provided, all project backups will be listed. |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
how are you managing list all backups for a project using this endpoint: https://api.pinecone.io/backups?
https://docs.pinecone.io/reference/api/2025-04/control-plane/list_project_backups
Since this endpoint is to list all backups for an index using https://api.pinecone.io/indexes/{index_name}/backups.
https://docs.pinecone.io/reference/api/2025-04/control-plane/list_index_backups
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Isn't indexName a required field since its a path param?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Good catch. There are two different resources for listing backups: https://github.com/pinecone-io/apis/blob/888bea7d82dc40f9c9252b80fb1f6de5b3f36ed1/src/release/db/control/control.yaml#L42-L57
GET /indexes/{index_name}/backups
- index levelGET /backups
- project level
I just combined the two to prevent having two different methods, and instead allow the user to determine which they're targeting through the optionality of indexName
. It calls the different operations under the hood: https://github.com/pinecone-io/pinecone-ts-client/pull/342/files#diff-82e31a389ec77273344d96a2e3f0ea165d1f6f8b7f0fbaf9fe55a42c489fbf5eR29-R32
It does seem like there's a problem with the spec (or possibly implementation on the server) here because limit and paginationToken are not defined as arguments to the listProjectBackups operation. We should follow up on that.
// } | ||
``` | ||
|
||
`listBackups` lists all the backups for a specific index, or your entire project. If an `indexName` is provided, only |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Same as before, isn't indexName a required field since its a path parameter?
## Problem The Backups & Restore feature was added in the `2025-04` API, and needs to be implemented in the client. Documentation: https://docs.pinecone.io/guides/manage-data/backups-overview ## Solution Backups and Restore functionality adds a number of new operations / methods under the Pinecone class. Add new action functions and unit tests for the following: - `createBackup` - `createIndexFromBackup` - `describeBackup` - `describeRestoreJob` - `listBackups` - `listRestoreJobs` - `deleteBackup` This PR also updates the `README` to include documentation around the feature, along with doc comments inline for all new methods, types, etc. We skipped integration tests for this feature for the moment since it can take some time for a restoration job to complete. I've added detailed unit test coverage for basic piping between the interface and underlying OpenAPI internals, plus validation and error handling. I've also done a fair bit of manual testing which I'll share below: ``` > await client.createBackup({ indexName: 'local-utility-index', name: 'my-backup', description: 'this is my backup' }) { backupId: '3df86526-0edf-41b0-a555-513effa2146d', sourceIndexName: 'local-utility-index', sourceIndexId: '1d18673a-ed0e-4cdc-946f-541560367abf', name: 'my-backup', description: 'this is my backup', status: 'Initializing', cloud: 'aws', region: 'us-east-1', dimension: undefined, metric: undefined, recordCount: undefined, namespaceCount: undefined, sizeBytes: undefined, tags: {}, createdAt: '2025-05-08T14:52:00.236917882Z' } > await client.describeBackup('3df86526-0edf-41b0-a555-513effa2146d') { backupId: '3df86526-0edf-41b0-a555-513effa2146d', sourceIndexName: 'local-utility-index', sourceIndexId: '1d18673a-ed0e-4cdc-946f-541560367abf', name: 'my-backup', description: 'this is my backup', status: 'Ready', cloud: 'aws', region: 'us-east-1', dimension: 5, metric: undefined, recordCount: 0, namespaceCount: 2, sizeBytes: 66877, tags: {}, createdAt: '2025-05-08T14:52:00.236989Z' } > await client.listBackups() { data: [ { backupId: '3df86526-0edf-41b0-a555-513effa2146d', sourceIndexName: 'local-utility-index', sourceIndexId: '1d18673a-ed0e-4cdc-946f-541560367abf', name: 'my-backup', description: 'this is my backup', status: 'Ready', cloud: 'aws', region: 'us-east-1', dimension: 5, metric: undefined, recordCount: 0, namespaceCount: 2, sizeBytes: 66877, tags: {}, createdAt: '2025-05-08T14:52:00.236989Z' }, ... ], pagination: undefined } > await client.createIndexFromBackup({ backupId: '11450b9f-96e5-47e5-9186-03f346b1f385', name: 'my-restored-index' }) { restoreJobId: '3de68445-77bf-4a74-9790-440523b5213b', indexId: '8987dd9f-f3e4-49a3-bbd2-3d73cb9c2810' } > await client.listRestoreJobs() { data: [ { restoreJobId: '3de68445-77bf-4a74-9790-440523b5213b', backupId: '11450b9f-96e5-47e5-9186-03f346b1f385', targetIndexName: 'my-restored-index', targetIndexId: '8987dd9f-f3e4-49a3-bbd2-3d73cb9c2810', status: 'Pending', createdAt: 2025-05-08T14:55:12.749Z, completedAt: undefined, percentComplete: undefined }, ... ], pagination: undefined } ``` ## Type of Change - [ ] Bug fix (non-breaking change which fixes an issue) - [X] New feature (non-breaking change which adds functionality) - [ ] Breaking change (fix or feature that would cause existing functionality to not work as expected) - [ ] This change requires a documentation update - [ ] Infrastructure change (CI configs, etc) - [ ] Non-code change (docs, etc) - [ ] None of the above: (explain here) ## Test Plan CI - external app test, integration tests, unit tests For manual testing, you can pull this branch down and run things locally with `npm run repl`. Set your `PINECONE_API_KEY` as an env variable, and then run the following: From the top level of the `pinecone-ts-client` repo: ```bash # this utility creates a new index and seeds some namespaces with data npx tsx ./utils/generateAndSeedIndex.ts npm run repl await init() await client.createBackup({ indexName: 'local-utility-index' }) await client.describeBackup('backup-id') await client.listBackups() await client.createIndexFromBackup({ backupId: 'backup-id' }) # etc ``` --- - To see the specific tasks where the Asana app for GitHub is being used, see below: - https://app.asana.com/0/0/1209571416750577
Problem
The Backups & Restore feature was added in the
2025-04
API, and needs to be implemented in the client.Documentation: https://docs.pinecone.io/guides/manage-data/backups-overview
Solution
Backups and Restore functionality adds a number of new operations / methods under the Pinecone class.
Add new action functions and unit tests for the following:
createBackup
createIndexFromBackup
describeBackup
describeRestoreJob
listBackups
listRestoreJobs
deleteBackup
This PR also updates the
README
to include documentation around the feature, along with doc comments inline for all new methods, types, etc.We skipped integration tests for this feature for the moment since it can take some time for a restoration job to complete. I've added detailed unit test coverage for basic piping between the interface and underlying OpenAPI internals, plus validation and error handling.
I've also done a fair bit of manual testing which I'll share below:
Type of Change
Test Plan
CI - external app test, integration tests, unit tests
For manual testing, you can pull this branch down and run things locally with
npm run repl
. Set yourPINECONE_API_KEY
as an env variable, and then run the following:From the top level of the
pinecone-ts-client
repo: